An empirical testing of user stereotypes of information retrieval systems

Stereotyping is a technique used in many information systems to represent user groups and/or to generate initial individual user models. However, there has been a lack of evidence on the accuracy of their use in representing users. We propose a formal evaluation method to test the accuracy or homogeneity of the stereotypes that are based on users explicit characteristics. Using the method, the results of an empirical testing on 11 common user stereotypes of information retrieval (IR) systems are reported. The participants memberships in the stereotypes were predicted using discriminant analysis, based on their IR knowledge. The actual membership and the predicted membership of each stereotype were compared. The data show that ‘‘librarians/IR professionals’’ is an accurate stereotype in representing its members, while some others, such as ‘‘undergraduate students’’ and ‘‘social sciences/humanities’’ users, are not accurate stereotypes. The data also demonstrate that based on the user s IR knowledge a stereotype can be made more accurate or homogeneous. The results show the promise that our method can help better detect the differences among stereotype members, and help with better stereotype design and user modeling. We assume that accurate stereotypes have better performance in user modeling and thus the system performance. Limitations and future directions of the study are discussed. 2004 Elsevier Ltd. All rights reserved.